Field of the Invention
One or more embodiments pertain to the field of event detection and tagging through use of sensors and media to detect events found in motion capture data, and/or media such as posts in a social media site and/or other sensors including but not limited to one or more of inertial, i.e., that detect orientation, position, velocity, acceleration, angular velocity, angular acceleration, or physical sensors, environmental sensors, chemical sensors and physiological sensors, i.e., electromagnetic field, temperature, humidity, wind, pressure, elevation, light, sound, heart rate, etc. Embodiments also enable motion capture data analysis and displaying information based on events recognized within the motion capture data or within motion analysis data associated with a user, or piece of equipment and/or based on previous motion analysis data from the user or other user(s) and/or piece of equipment. More particularly, but not by way of limitation, one or more embodiments enable a system that enables intelligent synchronization and transfer of curated event videos, i.e., generally concise event videos, synchronized with motion data from motion capture sensor(s) coupled with a user or piece of equipment. Greatly saves storage and increases upload speed by only saving or sending/obtaining or transferring relevant portions of media, e.g., uploading event videos instead of larger text, audio, image, video information with unwanted information. Creates highlight reels filtered by metrics and can sort by metric. Integrates with multiple sensors to save event data even if other sensors do not detect the event. Events may be correlated and/or otherwise synchronized with image(s) or video, as the events happen or at a later time based on location and/or time of the event or both, for example on a mobile device, which may include a camera, glasses with camera(s) and/or having a processor, mobile devices with camera(s), or on a remote server, and as captured from internal/external camera(s) or nanny cam, for example to enable saving video of the event, such as the first steps of a child, violent shaking events, sporting, military or other motion events including concussions, or falling events associated with an elderly person and for example discarding non-event related video data, to greatly reduce storage requirements for event videos. The system may automatically generate tags for events based on analysis of sensor data; tags may also be generated based on analysis of social media site postings describing the event. The system may use the combination of sensor data and media for example from social media sites to not only detect, confirm and publish events and curate media to provide concise versions of the events, but also determine whether an event is valid or invalid or represents fake news. One or more embodiments may be utilized to analyze multiple social media posts, or threads that are unknown across “friends” to determine events and/or provide emergency notifications, for example to flash all mobile device screens in case of a local emergency or terrorist attack, detected, confirmed and published by an embodiment of the invention.
Description of the Related Art
Existing systems do not utilize sensor data such as inertial data, i.e., motion capture data, including one or more of orientation, position, velocity, acceleration, angular velocity, angular acceleration, or other sensors such as physical, environmental, chemical and physiological sensors, i.e., electromagnetic field, temperature, humidity, wind, pressure, elevation, light, sound, heart rate, etc., to detect, confirm events, or public, i.e., post events, or differentiate similar types of motion events to determine the type of equipment or activity or quality of the event, such as how proficient a user is at a certain activity. Known systems do not curate or otherwise provide concise versions of text, images, (or 360 images), video, (or 360 video), sound or virtual reality for events and post the results to social networks using motion or other sensor data, for example in a dedicated feed. Known systems do not post or filter to social media sites for example using any other filter besides location and time and the text in the social media posts for example. There are no known systems that also use motion or other sensor data to define and event, eliminate false positive events, post true events, and/or correlate the events with social media to confirm the events, or post the events in a particular channel for example. Known systems do not use the combination of sensor data and media for example from social media sites to confirm events and do not curate media to provide concise versions of the events, and also do not determine whether an event is valid or invalid or represents fake news. Known systems do not analyze multiple social media posts, or threads that are unknown across “friends” to determine events, for example in combination with any sensor data and do not provide emergency notifications for example flash all screens, such as smart glasses screens or mobile device screens, etc., in case of a local emergency or terrorist attack.
Existing motion capture systems process and potentially store enormous amounts of data with respect to the actual events of interest. For example, known systems capture accelerometer data from sensors coupled to a user or piece of equipment and analyze or monitor movement. These systems do not intelligently confirm events using multiple disparate types of sensors or social media or other non-sensor based information, including postings to determine whether an event has actually occurred, or not, such as fake news, or what type of equipment or what type of activity has occurred. In these scenarios, thousands or millions of motion capture samples are associated with the user at rest or not moving in a manner that is related to a particular event that the existing systems are attempting to analyze. For example, if monitoring a football player, a large amount of motion data is not related to a concussion event, for a baby, a large amount of motion data is not related in general to a shaking event or non-motion event such as sudden infant death syndrome (SIDS), for a golfer, a large amount of motion data captured by a sensor mounted on the player's golf club is of low acceleration value, e.g., associated with the player standing or waiting for a play or otherwise not moving or accelerating in a manner of interest. Hence, capturing, transferring and storing non-event related data increases requirements for power, bandwidth and memory.
In addition, video capture of a user performing some type of motion may include even larger amounts of data, much of which has nothing to do with an actual event, such as a swing of a baseball bat or home run. There are no known systems that automatically curate or otherwise trim video, e.g., save event related video or even discard non-event related video, for example by uploading for example only the pertinent event video as determined by a sensor and/or motion capture sensor, without uploading the entire raw videos, to generate smaller media segments, i.e., text, audio, image or video segments that correspond to the events that occur in the media, e.g., video and for example as detected through analysis of the motion capture data.
Some systems that are related to monitoring impacts are focused on linear acceleration related impacts. These systems are unable to monitor rotational accelerations or velocities and are therefore unable to detect certain types of events that may produce concussions. In addition, many of these types of systems do not produce event related, connectionless messages for low power and longevity considerations. Hence, these systems are limited in their use based on their lack of robust characteristics.
Known systems also do not contemplate data mining of events within motion data to form a representation of a particular movement, for example a swing of an average player or average professional player level, or any player level based on a function of events recognized within previously stored motion data. Thus, it is difficult and time consuming and requires manual labor to find, trim and designate particular motion related events for use in virtual reality for example. Hence, current systems do not easily enable a particular user to play against a previously stored motion event of the same user or other user along with a historical player for example. Furthermore, known systems do not take into account cumulative impacts, and for example with respect to data mined information related to concussions, to determine if a series of impacts may lead to impaired brain function over time. No known systems integrate media and sensor data and determine, confirm and publish events, or curated events based on the combination of media and sensor data.
Other types of motion capture systems include video systems that are directed at analyzing and teaching body mechanics. These systems are based on video recording of an athlete and analysis of the recorded video of an athlete. This technique has various limitations including inaccurate and inconsistent subjective analysis based on video for example. Another technique includes motion analysis, for example using at least two cameras to capture three-dimensional points of movement associated with an athlete. Known implementations utilize a stationary multi-camera system that is not portable and thus cannot be utilized outside of the environment where the system is installed, for example during an athletic event such as a golf tournament, football game or to monitor a child or elderly person. In general video based systems do not also utilize digital motion capture data from sensors on the object undergoing motion since they are directed at obtaining and analyzing images having visual markers instead of electronic sensors. These fixed installations are extremely expensive as well. Such prior techniques are summarized in U.S. Pat. No. 7,264,554, filed 26 Jan. 2006, which claims the benefit of U.S. Provisional Patent Application Ser. No. 60/647,751 filed 26 Jan. 2005, the specifications of which are both hereby incorporated herein by reference. Both disclosures are to the same inventor of the subject matter of the instant application.
Regardless of the motion capture data obtained, the data is generally analyzed on a per user or per swing basis that does not contemplate processing on a mobile phone, so that a user would only buy a motion capture sensor and an “app” for a pre-existing mobile phone. In addition, existing solutions do not contemplate mobile use, analysis and messaging and/or comparison to or use of previously stored motion capture data from the user or other users or data mining of large data sets of motion capture data, for example to obtain or create motion capture data associated with a group of users, for example professional golfers, tennis players, baseball players or players of any other sport to provide events associated with a “professional level” average or exceptional virtual reality opponent. To summarize, motion capture data is generally used for immediate monitoring or sports performance feedback and generally has had limited and/or primitive use in other fields.
Known motion capture systems generally utilize several passive or active markers or several sensors. There are no known systems that utilize as little as one visual marker or sensor and an app that for example executes on a mobile device that a user already owns, to analyze and display motion capture data associated with a user and/or piece of equipment. The data is generally analyzed in a laboratory on a per user or per swing basis and is not used for any other purpose besides motion analysis or representation of motion of that particular user and is generally not subjected to data mining.
There are no known systems that allow for motion capture elements such as wireless sensors to seamlessly integrate or otherwise couple with a user or shoes, gloves, shirts, pants, belts, or other equipment, such as a baseball bat, tennis racquet, golf club, mouth piece for a boxer, football or soccer player, or protective mouthpiece utilized in any other contact sport for local analysis or later analysis in such a small format that the user is not aware that the sensors are located in or on these items. There are no known systems that provide seamless mounts, for example in the weight port of a golf club or at the end shaft near the handle so as to provide a wireless golf club, configured to capture motion data. Data derived from existing sensors is not saved in a database for a large number of events and is not used relative to anything but the performance at which the motion capture data was acquired.
In addition, for sports that utilize a piece of equipment and a ball, there are no known portable systems that allow the user to obtain immediate visual feedback regarding ball flight distance, swing speed, swing efficiency of the piece of equipment or how centered an impact of the ball is, i.e., where on the piece of equipment the collision of the ball has taken place. These systems do not allow for user's to play games with the motion capture data acquired from other users, or historical players, or from their own previous performances. Known systems do not allow for data mining motion capture data from a large number of swings to suggest or allow the searching for better or optimal equipment to match a user's motion capture data and do not enable original equipment manufacturers (OEMs) to make business decisions, e.g., improve their products, compare their products to other manufacturers, up-sell products or contact users that may purchase different or more profitable products.
In addition, there are no known systems that utilize motion capture data mining for equipment fitting and subsequent point-of-sale decision making for instantaneous purchasing of equipment that fits an athlete. Furthermore, no known systems allow for custom order fulfillment such as assemble-to-order (ATO) for custom order fulfillment of sporting equipment, for example equipment that is built to customer specifications based on motion capture data mining, and shipped to the customer to complete the point of sales process, for example during play or virtual reality play. Known systems do not publish any of this information on social media sites.
In addition, there are no known systems that use a mobile device and RFID tags for passive compliance and monitoring applications.
There are no known systems that enable data mining for a large number of users related to their motion or motion of associated equipment to find patterns in the data that allows for business strategies to be determined based on heretofore undiscovered patterns related to motion. There are no known systems that enable obtain payment from OEMs, medical professionals, gaming companies or other end users to allow data mining of motion data.
Known systems such as Lokshin, United States Patent Publication No. 20130346013, published 26 Dec. 2013 and 2013033054 published 12 Dec. 2013 for example do not contemplate uploading only the pertinent videos that occur during event, but rather upload large videos that are later synchronized. Both Lokshin references does not contemplate a motion capture sensor commanding a camera to alter camera parameters on-the-fly based on the event, to provide increased frame rate for slow motion for example during the event video capture, and do not contemplate changing playback parameters during a portion of a video corresponding to an event. The references also do not contemplate generation of highlight reels where multiple cameras may capture an event, for example from a different angle and do not contemplate automatic selection of the best video for a given event. In addition, the references do not contemplate a multi-sensor environment where other sensors may not observe or otherwise detect an event, while the sensor data is still valuable for obtaining metrics, and hence the references do not teach saving event data on other sensors after one sensor has identified an event.
Associating one or more tags with events is often useful for event analysis, filtering, and categorizing. Tags may for example indicate the players involved in an event, the type of action, and the result of an action (such as a score). Known systems rely on manual tagging of events by human operators who review event videos and event data. For example, there are existing systems for coaches to tag videos of sporting events or practices, for example to review a team's performance or for scouting reports. There are also systems for sports broadcasting that manually tag video events with players or actions. There are no known systems that analyze data from motion sensors, and media, e.g., video, radar, or other sensors to automatically select one or more tags for an event based on the data. An automatic event tagging system would provide a significant labor saving over the current manual tagging methods, and would provide valuable information for subsequent event retrieval and analysis. Known systems are unable to detect, confirm and publish events based on sensor data and media since they do not integrate the information obtained from these disparate sources.
For at least the limitations described above there is a need for an event detection, confirmation and publication system that integrates sensor data and social media.